Understanding the Limitations of Mathematical Reasoning in Large Language Models

Sent by Markus. https://arxiv.org/abs/2410.05229

"GSM-Symbolic"

GSM-Symbolic, an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions.

the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark.

their performance significantly deteriorates as the number of clauses in a question increases.